In recent years, advanced Web services represented by Web mail, social networks, and the like have rapidly come into widespread use. In such a Web service, when a Web server receives a Web page display request from a user, the Web server queries a backend storage of required data on individual texts, images, movies, and the like one by one, acquires the data, and transmits the data as a response to the user.
As such a storage, conventional file-system-based databases are used, but recent years have seen increasing use of object storages, in which unique URLs are specified to access content data, and which thus have a high compatibility with a contents access method over the Internet. Such an object storage employs a key-value store based I/F, with which a variable-length value is accessed with a unique variable-length key from a client device such as a Web server over Ethernet. In the case of Web servers, in general, the key is a URL and the value is content data. As to the key, an object ID is not variable-length but fixed-length, but basic principles are similar. Well-known systems include Ceph, Openstack, and the like as free software. In addition, as a storage device, an HDD or an SSD is used.
When an HDD or an SSD is used in combination with KVS (Key-Value Store) processing, owing to its poor random writing performance, a method called log structured writing is often used. In the log structured writing, couples of keys and values to be written (hereafter, described as key-values) are sequentially written on storage addresses in a receiving order of write requests. In the case of KVS processing using a hash table, a head address at which a key-value is written is saved to each entry in the hash table. If another writing of a key-value for the same key occurs, the address of a key-value that is written later is saved, and the key-value that is written later is used. The key-value that is written previously (or a key-value that is intentionally deleted) is overwritten with data subsequent on the address (the subsequent data is moved to the address at which the key-value existed) by garbage collection executed in the background, and a free space is increased accordingly.
Owing to the characteristics of the log structured writing, writing of one key-value cannot shift to writing of another key-value in the middle. For this reason, a key-value being written needs to be written to the last. Therefore, in the case where a key-value to be written is received over a network such as Ethernet, in particular in the case where the key-value is divided into a plurality of packets, writing the key-value in a storage cannot be started until the reception of all the packets forming the key-value is completely finished. The reason is that there is a risk that the reception of key-values stops in the middle due to network failure or a failure on a client device side. In addition, a network is shared among a plurality of communication connections (e.g., TCP connections), and a plurality of key-values may be simultaneously received from one or more client devices. In that case, a bandwidth occupied by a single communication connection is only a portion of the entire network bandwidth, and thus if receiving and writing are simultaneously performed, a writing speed is limited according to a speed of the occupied portion of network bandwidth. Therefore, from a viewpoint of efficient utilization of a storage performance, it is better to write key-values at once after the reception of the key-values is completely finished.
In the case where, in such a manner, the operation is to write key-values in a storage after the reception of the key-values is completely finished, a large-sized receiving buffer need to be prepared. For example, suppose that accesses occur from 1000 client devices (e.g., Web servers), assuming a large-scale data center or the like. When each client device is formed by, for example, two sockets (two CPUs), and each CPU includes 12 physical cores, this makes 24 threads per CPU virtual threading and 48 threads per client device, and the number of threads of all the client devices is 48,000. Assuming that a communication connection is made with each of these threads, and 1 MB of data, for example, is received from each of these communication connections simultaneously, this requires as much as a receiving buffer of 0.5 MB (average from 0 to 1 MB)×48000 connections=24 GB. While this is the case where the pieces of data are simultaneously received, which may be a rare case as an actual environment, the design for a memory may be made taking such a worst case into consideration.
In addition, in a method of securing a buffer memory in a fixed manner, it is needed to allocate buffers of a maximum data size for all the connections. For example, if the maximum data size is 1 MB, this requires 48 GB, which is twice as much as the above. In addition, if, during writing in the storage after the reception, the next reception is simultaneously received using, for example, a double buffer, this requires 96 GB of capacity, which is further twice as much.
Since the KVS processing is a basic function, this may be implemented as, for example, dedicated hardware logic. Since the hardware logic is not very good at dynamic memory allocation, the above-described fixed memory allocation may be performed, and the above double buffer configuration may be used for the pursuit of further performance. For this reason, the above-described 96 GB of capacity can actually come true in such a case. Such a buffer is typically implemented by a DRAM, and this results in a significant increase in cost.